docs: define backend plan split and SDS state contract - #737

What is a "semantic DAG"? Is this the output of ASAPPLanner? How is this different from SummaryMaintenanceLifecyclePlan?

“semantic DAG” means the selected post-ASAP computation DAG produced by ASAPPlanner.
SummaryMaintenanceLifecyclePlan means after a post-ASAP DAG being generated, some logic of summary maintainance will output the plan for how a summary is maintiend, incremental vs built from data at rest. So these are two steps currently in the code.

I didn't understand the relationship between Semantic DAG and SummaryMaintenanceLifecyclePlan. Also, what is ASAPQuery-backend inputting from ASAPPlanner? One of these or both?

this relationship will be answered in ProjectASAP/ASAPPlanner#445

The backend uses both, at different stages:

1. Selected post-ASAP DAG: The backend’s selection adapter obtains a Rc root and passes it to physical compilation as QueryCompilationInput.selected_plan_root. This describes the
   selected computation. Code

2. Maintenance lifecycle decisions: During physical compilation, select_lifecycle() calls Planner’s plan_summary_maintenance_lifecycles() with the selected summary node, workload demand,
   capabilities, and cost evidence. That returns a SummaryMaintenanceLifecyclePlan. The backend extracts its selected lifecycle guarantee, window framework, implementation identity, and
   costing information into backend configuration. Code

So the current flow is:

Select post-ASAP computation
→ pass selected root to backend compiler
→ compiler calls Planner for summary lifecycle decisions
→ combine computation and selected maintenance decisions
→ generate backend plans

The lifecycle plan refers to the summary computation it is planning maintenance for. It does not replace the DAG, and the backend does not simply execute the lifecycle-plan object directly.

Following up on my explanation above: that describes the current backend call sequence. Based on ASAPPlanner #445, I propose moving lifecycle-aware selection before physical compilation and passing its result directly to the compiler.

Current implementation

Select complete Post-ASAP DAG
    -> backend physical compiler receives selected_plan_root
         -> extracts summary producers
         -> calls Planner lifecycle API for selected producers
         -> extracts lifecycle/window decisions
         -> combines those decisions with the complete query DAG
         -> generates backend physical plans

The current compiler calls select_lifecycle(..., &selected.node, ...) on extracted producers (code). Those producer-local lifecycle results do not necessarily contain downstream query readouts, so the compiler still needs the separate complete query root.

Proposed integration

PlanningWorkload + evidence + capabilities
    -> ASAPPlanner
    -> PlanSpace
    -> lifecycle-aware selection and materialization
    -> SummaryMaintenanceLifecyclePlan
         - complete selected Post-ASAP root
         - maintenance decisions for its summary producers
         - summary-versus-raw recomputation decision
    -> backend physical compiler validates and binds the selected result
         -> PrecomputePlan
         -> QueryPlan
         -> Summary Catalog definitions
    -> installation and execution

PlanSpace remains Planner's canonical logical output; lifecycle-aware selection is a helper over that output, as described in #445. The compiler still needs physical implementation, schema, storage and installation context. The simplification concerns the computation/lifecycle handoff, not removal of backend responsibilities.

Relative to the current PR #737 design, this is a smaller change: the document already requires the selected DAG plus lifecycle commitments. The proposal makes their handoff one associated result instead of independently supplied artifacts. The PrecomputePlan/QueryPlan split, catalog definitions and state references remain applicable.

The essential condition is that the result retains the complete query root, including readouts and remaining query operations. For multiple queries, preserve query-to-root associations and shared producer identity across the results so compilation does not duplicate maintenance. If raw recomputation is selected, do not unconditionally install summary producers.

This is a target integration proposal, not an interface the current backend already implements. It requires moving the lifecycle-selection call boundary and preserving those associations, not merely changing a parameter type.

Pls add a note that this is confusing and must be changed. Related #740

I do not understand these. Is there documentation?

Examples are helpful thank you. What does range mean?

range: 5m means the logical input window summarized by the KLL. For an evaluation at time T, it includes samples with timestamps in (T - 5m, T], grouped by service.

So basically it's equal to the size of the time window (tumbling or sliding) used to generate the KLL instances?

yes

These concepts are also present in ASAPPlanner right? Wondering if these have been described there

yes

How is this catalog actually used?

Catalog stores the semantics / description of SDS (changed less often), the summary store/sketch store stores the SDS instances payloads (changed per instance).

Catalog is metadata store

I understand what it is. I am curious, how it is used right now. Is it used right now?

catalog is being used rn.

How it's being used --

• Compilation: The physical compiler constructs SummaryCatalog from the selected precompute configurations, binds PrecomputePlan to it, and validates QueryPlan against it. Compiler code
• Runtime installation: When the startup physical plan contains a catalog, the data plane installs it into SketchStore. Startup code
• Query execution: One concrete consumer is MetricsQL per-series readout: it looks up the bound definition in the catalog, resolves its data descriptor, and restores the metric’s name
  label. If that metadata is unavailable, that path requests fallback. Readout code

The current catalog contains summary_descriptors, data_descriptors, and a materializations map. The simplified catalog described in this PR is a proposed change to that existing
representation—not the introduction of a previously unused catalog. Current type

Confused by this. I dont understand the meaning. Also, how is this "Materialization" related to the discussion at #736 (comment)

• SummaryDefinition: summarize request_latency_seconds by service over five minutes using KLL with k=200.
• SummaryStateInstance: one concrete stored result from that producer, such as the summary for service=api covering (12:00, 12:05].

Agreed—the standalone catalog Materialization object was an over-abstraction. I have removed it from the proposed design and examples rather than introducing another name for the same layer.

Its information now belongs to the objects that use it:

The resulting flow is simply:

PrecomputePlan: Build KLL -> Write state slot S
QueryPlan:     Read state slot S -> Estimate p99
Catalog:       SummaryDefinition referenced by both bindings

The state slot is only a join key within a plan version, not a new catalog object with an independent lifecycle. The compiler emits both bindings from one decision and validates their agreement before installation.

Regarding #736: BackendNodeBinding::Materialization remains the existing node-placement marker meaning “store this node's output.” It does not require a separate catalog Materialization object. I have made that distinction explicit.

The docs now remove the proposed materializations collection, update the diagrams and examples, and describe migration through versioned adapters: map existing stored-output identities to slots, retain payload locators, and preserve the format/partition constraints in reader/writer bindings. Existing persisted IDs and wire fields must not be silently renamed or reinterpreted. This PR remains a design-document change; runtime migration is follow-up implementation work.

Okay thanks

I did not understand the purpose of this.

statereference is how plan can reference the summarystateinstance.

Still confused on this. Why do we need this concept?

Good question. I used “lifecycle commitment” only as shorthand for the selected, executable maintenance decision for one summary producer: the selected SummaryMaintenanceLifecycleGuarantee together with its concrete schedule and retention binding. This distinguishes the per-producer decision from the SummaryMaintenanceLifecyclePlan, which contains the DAG, all producer deployments, alternatives, and workload/cost context. It is not a new model or API type. If the shorthand obscures that distinction, I can use “selected deployment guarantee and schedule/retention” throughout instead.

Updated the glossary, integration design, migration plan, and conceptual YAML example to use “selected deployment guarantee and schedule/retention” instead. This is now commit ac855b2.

Did not understand last sentence on missing guarantee

Agreed; that sentence was too compressed. I rewrote it in ea3226c. In the Planner type, a deployment’s guarantee is optional (None when no alternative is selectable). For that producer, the backend has no selected maintenance mode/schedule to implement, so it must not invent one. If a query needs that stored state, it must use an explicit supported fallback or the plan must be rejected.

Very confused by this. Can you give an example situation where if I do not have this concept, there is some issue with correctness or performance or something else?

I saw the compiler input example, but still do not understand this concept.

You are right that the old “lifecycle commitment” wording made this sound like a separate concept. I removed that term and added a concrete example in 118e363. Suppose two queries share a five-minute KLL summary and ask for a result every minute. The selected deployment says to rebuild at each minute boundary and retain completed snapshots for ten minutes. The DAG identifies the shared computation, but by itself does not specify those maintenance decisions. If the backend independently rebuilds only every five minutes, four of five requested endpoints have no matching state; reusing an older snapshot as if it covered the requested interval would change the answer. So the need is to carry and validate the existing Planner-selected guarantee and schedule/retention, not introduce a new model or type.

For the edges to SummaryDefinition catalog, which are reads and which are writes?
Related, are you saying that if I think of an SDS, the SummaryStore only stores SDS payload, while other metadata stays in the SummaryDefinition metadata catalog?

Good catch; the arrows were ambiguous. I labeled them in 1effeb5. The compiler/catalog authority writes the SummaryDefinition when installing a plan. The PrecomputePlan and QueryPlan bindings reference that definition; those edges mean catalog reads for validation at installation, not runtime catalog writes or serving-time searches. At runtime, PrecomputePlan writes encoded summary payload bytes to the Summary Store and publishes partition/coverage/format/readiness/location as SummaryStateInstance metadata in the runtime inventory. QueryPlan resolves a ready instance from that inventory and reads its payload. So SDS is the contract across the definition catalog, installed plan bindings, runtime instance metadata, and payload store; the other metadata is not all in SummaryDefinition.

Better names for these might be -- SummaryPayloadStore and SummaryMetadataStore ?

good idea

This is confusing. Later, you describe the Catalog as storing SummaryDefinition, not metadata?

Meaning of &shared_read and *shared_read ?

Correction: those were YAML anchor/alias syntax and distracted from the design. I removed both and expanded the complete read_state binding under q50 and q99.

Does this include both payload and metadata?

Correction: yes, conceptually. SummaryStateInstance means one complete logical SummaryStore entry: instance metadata plus its associated payload. An implementation may keep the bytes separately internally, but that does not create another architecture component.

what is this?

What does it mean for a plan to have state_instances? This YAML block seems to describe the metadata of a particular SDS instance.

Correction: runtime instances are not part of the plan. The example now has separate installed_plan and runtime_summary_store sections. The latter is explicitly observed runtime data produced after the PrecomputePlan writes state.

what is slot?

Correction: I renamed state_slot_id to stored_output_id and defined it as the plan-version-scoped binding ID for one persisted PrecomputePlan DAG output. The writer and QueryPlan readers use it to name the same output; it is not a memory slot or storage object.

What's the usecase for these?

The five-phase table mixed plan intent, build progress, and storage retirement without a current SDS use case. I replaced it with the one decision the DAG path needs now: a QueryPlan read is eligible only when its bound instance exists, the payload is committed, and version, definition, format, partition, and coverage match. Building/draining/retirement remain existing runtime concerns rather than new SDS states.

This might be the data structure but it's not exposing the semantics of a definition i.e. the stuff you wrote above like source and filter semantics, value expressions, etc.

what is endpoint east and west?